<div id="content_views" class="markdown_views prism-tomorrow-night-eighties">
                    <svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
                        <path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path>
                    </svg>
                    <h2><a name="t0"></a><a id="Urllib_0"></a>Urllib</h2>
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/939a69500d1bd192e2d866c19eaae037.png" alt="在这里插入图片描述"></p>
<h2><a name="t1"></a><a id="_3"></a>一、反爬手段</h2>
<ul><li><strong>User‐Agent</strong>：
  <ul><li>User Agent中文名为用户代理，简称 UA，它是一个特殊字符串头，使得服务器能够识别客户使用的操作系统及版本、CPU 类型、浏览器及版本、浏览器渲染引擎、浏览器语言、浏览器插件等。</li></ul> </li><li><strong>代理IP</strong>
  <ul><li>西次代理</li><li>快代理</li><li>什么是高匿名、匿名和透明代理？它们有什么区别？<br> 1.使用透明代理，对方服务器可以知道你使用了代理，并且也知道你的真实IP。<br> 2.使用匿名代理，对方服务器可以知道你使用了代理，但不知道你的真实IP。<br> 3.使用高匿名代理，对方服务器不知道你使用了代理，更不知道你的真实IP。</li></ul> </li><li><strong>验证码访问</strong><br> - 打码平台<br> - 云打码平台<br> - 超级🦅</li><li><strong>动态加载网页</strong> 网站返回的是js数据 并不是网页的真实数据
  <ul><li>selenium驱动真实的浏览器发送请求</li></ul> </li><li><strong>数据加密</strong>
  <ul><li>分析js代码</li></ul> </li></ul>
<hr>
<h3><a name="t2"></a><a id="urllib_23"></a>二、urllib库使用</h3>
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/1e04067c8d99974d92ad4ca0f9da9894.png" alt="在这里插入图片描述"></p>
<ul><li><strong>基本使用</strong><pre data-index="0" class="set-code-show prettyprint"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request
<span class="token comment"># 使用urllib获取百度首页源码</span>
url <span class="token operator">=</span> <span class="token string">'http://www.baidu.com'</span>

<span class="token comment"># 模拟浏览器发送请求</span>
resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>url<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>resp<span class="token punctuation">)</span>
context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li></ul></pre> </li></ul>
<hr>
<h3><a name="t3"></a><a id="_40"></a>三、请求对象的定制</h3>
<blockquote>
 <p><a href="https://so.csdn.net/so/search?q=UA&amp;spm=1001.2101.3001.7020" target="_blank" class="hl hl-1" data-report-click="{&quot;spm&quot;:&quot;1001.2101.3001.7020&quot;,&quot;dest&quot;:&quot;https://so.csdn.net/so/search?q=UA&amp;spm=1001.2101.3001.7020&quot;,&quot;extra&quot;:&quot;{\&quot;searchword\&quot;:\&quot;UA\&quot;}&quot;}" data-tit="UA" data-pretit="ua">UA</a>介绍：User Agent中文名为用户代理，简称 UA，它是一个特殊字符串头，使得服务器能够识别客户使用的操作系统及版本、CPU 类型、浏览器及版本。浏览器内核、浏览器渲染引擎、浏览器语言、浏览器插件等</p>
</blockquote>
<p>语法：<code>request = urllib.request.Request()</code></p>
<pre data-index="1" class="set-code-show prettyprint"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request

url <span class="token operator">=</span> <span class="token string">'https://www.baidu.com'</span>
headers<span class="token operator">=</span><span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span><span class="token string">"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36"</span>
<span class="token punctuation">}</span>

requestObj<span class="token operator">=</span>urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">,</span>headers<span class="token punctuation">)</span>

resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>requestObj<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li></ul></pre>
<hr>
<h3><a name="t4"></a><a id="_60"></a>四、编解码</h3>
<h4><a name="t5"></a><a id="1geturllibparsequote_61"></a>1、get请求方式：urllib.parse.quote（）</h4>
<pre data-index="2" class="set-code-show prettyprint"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request

searchWord <span class="token operator">=</span> urllib<span class="token punctuation">.</span>parse<span class="token punctuation">.</span>quote<span class="token punctuation">(</span><span class="token string">"蝙蝠侠"</span><span class="token punctuation">)</span>
url <span class="token operator">=</span> <span class="token string">'https://www.sogou.com/web?query='</span><span class="token operator">+</span> searchWord
headers<span class="token operator">=</span><span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span> <span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span>
<span class="token punctuation">}</span>

<span class="token comment"># print(url)</span>
requestObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>
resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>requestObj<span class="token punctuation">)</span>
context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li></ul></pre>
<hr>
<h4><a name="t6"></a><a id="2geturllibparseurlencode_80"></a>2、get请求方式：urllib.parse.urlencode（）</h4>
<pre data-index="3" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request

data <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">'query'</span><span class="token punctuation">:</span><span class="token string">'蝙蝠侠'</span><span class="token punctuation">,</span>
    <span class="token string">'w'</span><span class="token punctuation">:</span><span class="token string">'01019900'</span>
<span class="token punctuation">}</span>

searchWord <span class="token operator">=</span> urllib<span class="token punctuation">.</span>parse<span class="token punctuation">.</span>urlencode<span class="token punctuation">(</span>data<span class="token punctuation">)</span>
preUrl <span class="token operator">=</span> <span class="token string">'https://www.sogou.com/web?'</span>
url <span class="token operator">=</span> preUrl<span class="token operator">+</span>searchWord

requestObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token punctuation">,</span><span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">,</span>headers<span class="token operator">=</span><span class="token punctuation">{<!-- --></span><span class="token string">'User-Agent'</span><span class="token punctuation">:</span><span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span><span class="token punctuation">}</span><span class="token punctuation">)</span>

resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>requestObj<span class="token punctuation">)</span>
context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li></ul></pre>
<hr>
<h4><a name="t7"></a><a id="3post_102"></a>3、post请求方式</h4>
<pre data-index="4" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token comment">#eg:百度翻译</span>

<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
url <span class="token operator">=</span> <span class="token string">'https://fanyi.baidu.com/sug'</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">'user‐agent'</span><span class="token punctuation">:</span> 'Mozilla<span class="token operator">/</span><span class="token number">5.0</span> <span class="token punctuation">(</span>Windows NT <span class="token number">10.0</span><span class="token punctuation">;</span> Win64<span class="token punctuation">;</span> x64<span class="token punctuation">)</span> AppleWebKit<span class="token operator">/</span><span class="token number">537.36</span> <span class="token punctuation">(</span>KHTML<span class="token punctuation">,</span> like
Gecko<span class="token punctuation">)</span> Chrome<span class="token operator">/</span><span class="token number">74.0</span><span class="token number">.3729</span><span class="token number">.169</span> Safari<span class="token operator">/</span><span class="token number">537.36</span>'
<span class="token punctuation">}</span>
keyword <span class="token operator">=</span> <span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">'请输入您要查询的单词'</span><span class="token punctuation">)</span>
data <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">'kw'</span><span class="token punctuation">:</span>keyword
<span class="token punctuation">}</span>
data <span class="token operator">=</span> urllib<span class="token punctuation">.</span>parse<span class="token punctuation">.</span>urlencode<span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span><span class="token string">'utf‐8'</span><span class="token punctuation">)</span>
request <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">,</span>data<span class="token operator">=</span>data<span class="token punctuation">)</span>
response <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>request<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf‐8'</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li></ul></pre>
<ul><li><strong>百度详细翻译</strong></li></ul>
<pre data-index="5" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request
<span class="token keyword">import</span> json

url <span class="token operator">=</span> <span class="token string">'https://fanyi.baidu.com/v2transapi?from=zh&amp;to=en'</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span> <span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span><span class="token punctuation">,</span>
    <span class="token string">'Accept'</span><span class="token punctuation">:</span><span class="token string">'*/*'</span><span class="token punctuation">,</span>
    <span class="token comment"># 'Accept-Encoding':'gzip, deflate, br',</span>
    <span class="token string">'Accept-Language'</span><span class="token punctuation">:</span><span class="token string">'zh-CN,zh;q=0.9,en;q=0.8'</span><span class="token punctuation">,</span>
    <span class="token string">'Connection'</span><span class="token punctuation">:</span><span class="token string">'keep-alive'</span><span class="token punctuation">,</span>
    <span class="token string">'Content-Length'</span><span class="token punctuation">:</span><span class="token string">'148'</span><span class="token punctuation">,</span>
    <span class="token string">'Content-Type'</span><span class="token punctuation">:</span><span class="token string">'application/x-www-form-urlencoded; charset=UTF-8'</span><span class="token punctuation">,</span>
    <span class="token string">'Cookie'</span><span class="token punctuation">:</span><span class="token string">'BIDUPSID=E4EED4AD32F68A0029150518B71E6473; PSTM=1644069814; BAIDUID=E4EED4AD32F68A00C795D774149F4DCE:FG=1; __yjs_duid=1_03f1094640beee76272a007638c459db1644071304166; BDUSS=dsWE1lc0kxbWdZTHNLejFRanNJOVNvZjVQSjJaZnpFWFlIa2t6ZXAyYnhqaWxpSVFBQUFBJCQAAAAAAAAAAAEAAAA0MjAmb285OTU5MzE1NzYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPEBAmLxAQJiR1; BDUSS_BFESS=dsWE1lc0kxbWdZTHNLejFRanNJOVNvZjVQSjJaZnpFWFlIa2t6ZXAyYnhqaWxpSVFBQUFBJCQAAAAAAAAAAAEAAAA0MjAmb285OTU5MzE1NzYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPEBAmLxAQJiR1; FANYI_WORD_SWITCH=1; REALTIME_TRANS_SWITCH=1; HISTORY_SWITCH=1; SOUND_PREFER_SWITCH=1; SOUND_SPD_SWITCH=1; APPGUIDE_10_0_2=1; Hm_lvt_64ecd82404c51e03dc91cb9e8c025574=1644391469,1644461433; BDORZ=B490B5EBF6F3CD402E515D22BCDA1598; delPer=0; PSINO=5; BAIDUID_BFESS=0434881F591F38E40670B2C4E152E2CB:FG=1; H_PS_PSSID=34430_35104_31254_35489_34584_35490_35542_35797_35320_26350_35746; BA_HECTOR=0l0g008g802g8h8ljs1h09avc0r; Hm_lpvt_64ecd82404c51e03dc91cb9e8c025574=1644474869; ab_sr=1.0.1_NjFhODYxOTg3ODkzNmExOThhNGNmZjNkYWQ1MGUyYjgyNWEzMTMzOTcxM2IzY2Q1NDk3ZjE4YzgyZDhlNzIyNTdlYTBhZjY5MTFjOWRiYzgzYTg5ZGExNjQxNGVkM2U1YjQyNzVjMDc4N2M0YTVjYzcyYzAyODMyNjY0MDEyMTM0MTQ4NzllMmFiZTM3MDNlYzY3OTUzMTBmMWE2NzM2YmY5ZDQ5Nzk1NDMzMWI0MWI0ZTg4NDNkNTFmM2M4ZTFm'</span><span class="token punctuation">,</span>
    <span class="token string">'Host'</span><span class="token punctuation">:</span><span class="token string">'fanyi.baidu.com'</span><span class="token punctuation">,</span>
    <span class="token string">'Origin'</span><span class="token punctuation">:</span> <span class="token string">'https://fanyi.baidu.com'</span><span class="token punctuation">,</span>
    <span class="token string">'Referer'</span><span class="token punctuation">:</span> <span class="token string">'https://fanyi.baidu.com/translate?aldtype=16047&amp;query=&amp;keyfrom=baidu&amp;smartresult=dict&amp;lang=auto2zh'</span><span class="token punctuation">,</span>
    <span class="token string">'sec-ch-ua'</span><span class="token punctuation">:</span> <span class="token string">'"Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"'</span><span class="token punctuation">,</span>
    <span class="token string">'sec-ch-ua-mobile'</span><span class="token punctuation">:</span> <span class="token string">'?0'</span><span class="token punctuation">,</span>
    <span class="token string">'sec-ch-ua-platform'</span><span class="token punctuation">:</span> <span class="token string">"Windows"</span><span class="token punctuation">,</span>
    <span class="token string">'Sec-Fetch-Dest'</span><span class="token punctuation">:</span> <span class="token string">'empty'</span><span class="token punctuation">,</span>
    <span class="token string">'Sec-Fetch-Mode'</span><span class="token punctuation">:</span> <span class="token string">'cors'</span><span class="token punctuation">,</span>
    <span class="token string">'Sec-Fetch-Site'</span><span class="token punctuation">:</span> <span class="token string">'same-origin'</span><span class="token punctuation">,</span>
    <span class="token string">'X-Requested-With'</span><span class="token punctuation">:</span> <span class="token string">'XMLHttpRequest'</span>
<span class="token punctuation">}</span>
data <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">'from'</span><span class="token punctuation">:</span> <span class="token string">'zh'</span><span class="token punctuation">,</span>
    <span class="token string">'to'</span><span class="token punctuation">:</span> <span class="token string">'en'</span><span class="token punctuation">,</span>
    <span class="token string">'query'</span><span class="token punctuation">:</span> <span class="token string">'房子'</span><span class="token punctuation">,</span>
    <span class="token string">'transtype'</span><span class="token punctuation">:</span> <span class="token string">'realtime'</span><span class="token punctuation">,</span>
    <span class="token string">'simple_means_flag'</span><span class="token punctuation">:</span> <span class="token string">'3'</span><span class="token punctuation">,</span>
    <span class="token string">'sign'</span><span class="token punctuation">:</span> <span class="token string">'289459.35202'</span><span class="token punctuation">,</span>
    <span class="token string">'token'</span><span class="token punctuation">:</span> <span class="token string">'1ee0174682b16644fbbcde79861a56e5'</span><span class="token punctuation">,</span>
    <span class="token string">'domain'</span><span class="token punctuation">:</span> <span class="token string">'common'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>
key <span class="token operator">=</span> urllib<span class="token punctuation">.</span>parse<span class="token punctuation">.</span>urlencode<span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
requestObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token punctuation">,</span>key<span class="token punctuation">,</span>headers<span class="token punctuation">)</span>
resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>requestObj<span class="token punctuation">)</span>
context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
<span class="token builtin">str</span> <span class="token operator">=</span> json<span class="token punctuation">.</span>loads<span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li><li style="color: rgb(153, 153, 153);">18</li><li style="color: rgb(153, 153, 153);">19</li><li style="color: rgb(153, 153, 153);">20</li><li style="color: rgb(153, 153, 153);">21</li><li style="color: rgb(153, 153, 153);">22</li><li style="color: rgb(153, 153, 153);">23</li><li style="color: rgb(153, 153, 153);">24</li><li style="color: rgb(153, 153, 153);">25</li><li style="color: rgb(153, 153, 153);">26</li><li style="color: rgb(153, 153, 153);">27</li><li style="color: rgb(153, 153, 153);">28</li><li style="color: rgb(153, 153, 153);">29</li><li style="color: rgb(153, 153, 153);">30</li><li style="color: rgb(153, 153, 153);">31</li><li style="color: rgb(153, 153, 153);">32</li><li style="color: rgb(153, 153, 153);">33</li><li style="color: rgb(153, 153, 153);">34</li><li style="color: rgb(153, 153, 153);">35</li><li style="color: rgb(153, 153, 153);">36</li><li style="color: rgb(153, 153, 153);">37</li><li style="color: rgb(153, 153, 153);">38</li><li style="color: rgb(153, 153, 153);">39</li><li style="color: rgb(153, 153, 153);">40</li><li style="color: rgb(153, 153, 153);">41</li></ul></pre>
<hr>
<h3><a name="t8"></a><a id="ajaxget_169"></a>五、ajax的get请求</h3>
<ul><li><strong>get请求豆瓣电影前十页</strong></li></ul>
<pre data-index="6" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request
<span class="token keyword">import</span> json

<span class="token comment"># url = 'https://movie.douban.com/j/chart/top_list?type=19&amp;interval_id=100%3A90&amp;action=&amp;start='+page+'&amp;limit=20'</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span> <span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>


<span class="token keyword">def</span> <span class="token function">getjson</span><span class="token punctuation">(</span>page<span class="token punctuation">)</span><span class="token punctuation">:</span>
    url <span class="token operator">=</span> <span class="token string">'https://movie.douban.com/j/chart/top_list?type=19&amp;interval_id=100%3A90&amp;action=&amp;start='</span> <span class="token operator">+</span> <span class="token builtin">str</span><span class="token punctuation">(</span>
        <span class="token punctuation">(</span>page <span class="token operator">-</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">20</span><span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token string">'&amp;limit=20'</span>
    requestObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span> headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>
    resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>requestObj<span class="token punctuation">)</span>
    context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
    <span class="token keyword">return</span> context


<span class="token keyword">def</span> <span class="token function">download</span><span class="token punctuation">(</span>page<span class="token punctuation">,</span> context<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">"豆瓣电影_第"</span> <span class="token operator">+</span> <span class="token builtin">str</span><span class="token punctuation">(</span>page<span class="token punctuation">)</span> <span class="token operator">+</span> <span class="token string">"页.json"</span><span class="token punctuation">,</span> <span class="token string">'w'</span><span class="token punctuation">,</span> encoding<span class="token operator">=</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fp<span class="token punctuation">:</span>
        fp<span class="token punctuation">.</span>write<span class="token punctuation">(</span>context<span class="token punctuation">)</span>


<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
    start_page <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">"开始页"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    end_page <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">"结束页"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    <span class="token keyword">for</span> page <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span>start_page<span class="token punctuation">,</span> end_page <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        context <span class="token operator">=</span> getjson<span class="token punctuation">(</span>page<span class="token punctuation">)</span>
        <span class="token keyword">if</span> context <span class="token operator">!=</span> <span class="token string">'[]'</span><span class="token punctuation">:</span>
            download<span class="token punctuation">(</span>page<span class="token punctuation">,</span> context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li><li style="color: rgb(153, 153, 153);">18</li><li style="color: rgb(153, 153, 153);">19</li><li style="color: rgb(153, 153, 153);">20</li><li style="color: rgb(153, 153, 153);">21</li><li style="color: rgb(153, 153, 153);">22</li><li style="color: rgb(153, 153, 153);">23</li><li style="color: rgb(153, 153, 153);">24</li><li style="color: rgb(153, 153, 153);">25</li><li style="color: rgb(153, 153, 153);">26</li><li style="color: rgb(153, 153, 153);">27</li><li style="color: rgb(153, 153, 153);">28</li><li style="color: rgb(153, 153, 153);">29</li><li style="color: rgb(153, 153, 153);">30</li><li style="color: rgb(153, 153, 153);">31</li></ul></pre>
<hr>
<h3><a name="t9"></a><a id="ajaxpost_207"></a>六、ajax的post请求</h3>
<ul><li><strong>post请求肯德基温州门店信息</strong></li></ul>
<pre data-index="7" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request


<span class="token comment"># http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=cname</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span> <span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>
url <span class="token operator">=</span> <span class="token string">'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=cname'</span>


<span class="token keyword">def</span> <span class="token function">getSearchParams</span><span class="token punctuation">(</span>page<span class="token punctuation">,</span>city<span class="token punctuation">)</span><span class="token punctuation">:</span>
    data <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
        <span class="token string">'cname'</span><span class="token punctuation">:</span> city<span class="token punctuation">,</span>
        <span class="token string">'pageIndex'</span><span class="token punctuation">:</span> page<span class="token punctuation">,</span>
        <span class="token string">'pageSize'</span><span class="token punctuation">:</span> <span class="token string">'10'</span>
    <span class="token punctuation">}</span>
    searchParams <span class="token operator">=</span> urllib<span class="token punctuation">.</span>parse<span class="token punctuation">.</span>urlencode<span class="token punctuation">(</span>data<span class="token punctuation">)</span><span class="token punctuation">.</span>encode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>
    <span class="token keyword">return</span> searchParams


<span class="token keyword">def</span> <span class="token function">getContext</span><span class="token punctuation">(</span>searchParams<span class="token punctuation">)</span><span class="token punctuation">:</span>
    reqObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span> headers<span class="token operator">=</span>headers<span class="token punctuation">,</span> data<span class="token operator">=</span>searchParams<span class="token punctuation">)</span>
    resp <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>reqObj<span class="token punctuation">)</span>
    context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
    <span class="token keyword">return</span> context


<span class="token keyword">def</span> <span class="token function">download</span><span class="token punctuation">(</span>city<span class="token punctuation">,</span>index<span class="token punctuation">,</span>context<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">"肯德基门店信息_"</span> <span class="token operator">+</span> city <span class="token operator">+</span> <span class="token string">"_"</span> <span class="token operator">+</span> <span class="token builtin">str</span><span class="token punctuation">(</span>index<span class="token punctuation">)</span><span class="token operator">+</span><span class="token string">'.json'</span><span class="token punctuation">,</span> <span class="token string">'w'</span><span class="token punctuation">,</span> encoding<span class="token operator">=</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fp<span class="token punctuation">:</span>
        fp<span class="token punctuation">.</span>write<span class="token punctuation">(</span>context<span class="token punctuation">)</span>


<span class="token keyword">if</span> __name__ <span class="token operator">==</span> <span class="token string">'__main__'</span><span class="token punctuation">:</span>
    page <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">"爬取第几页？"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>
    city <span class="token operator">=</span> <span class="token builtin">input</span><span class="token punctuation">(</span><span class="token string">"什么城市？"</span><span class="token punctuation">)</span>
    <span class="token keyword">for</span> index <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span>page<span class="token operator">+</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        searchParams <span class="token operator">=</span> getSearchParams<span class="token punctuation">(</span>index<span class="token punctuation">,</span>city<span class="token punctuation">)</span>
        context <span class="token operator">=</span> getContext<span class="token punctuation">(</span>searchParams<span class="token punctuation">)</span>
        <span class="token keyword">if</span> context <span class="token operator">!=</span> <span class="token string">'[]'</span><span class="token punctuation">:</span>
            download<span class="token punctuation">(</span>city<span class="token punctuation">,</span>index<span class="token punctuation">,</span>context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li><li style="color: rgb(153, 153, 153);">18</li><li style="color: rgb(153, 153, 153);">19</li><li style="color: rgb(153, 153, 153);">20</li><li style="color: rgb(153, 153, 153);">21</li><li style="color: rgb(153, 153, 153);">22</li><li style="color: rgb(153, 153, 153);">23</li><li style="color: rgb(153, 153, 153);">24</li><li style="color: rgb(153, 153, 153);">25</li><li style="color: rgb(153, 153, 153);">26</li><li style="color: rgb(153, 153, 153);">27</li><li style="color: rgb(153, 153, 153);">28</li><li style="color: rgb(153, 153, 153);">29</li><li style="color: rgb(153, 153, 153);">30</li><li style="color: rgb(153, 153, 153);">31</li><li style="color: rgb(153, 153, 153);">32</li><li style="color: rgb(153, 153, 153);">33</li><li style="color: rgb(153, 153, 153);">34</li><li style="color: rgb(153, 153, 153);">35</li><li style="color: rgb(153, 153, 153);">36</li><li style="color: rgb(153, 153, 153);">37</li><li style="color: rgb(153, 153, 153);">38</li><li style="color: rgb(153, 153, 153);">39</li><li style="color: rgb(153, 153, 153);">40</li><li style="color: rgb(153, 153, 153);">41</li></ul></pre>
<hr>
<h3><a name="t10"></a><a id="URLErrorHTTPError_254"></a>七、URLError\HTTPError</h3>
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/546394b94a60e51514ede1d862b64b05.png" alt="在这里插入图片描述"></p>
<pre data-index="8" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;">eg<span class="token punctuation">:</span>
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>error
url <span class="token operator">=</span> <span class="token string">'https://blog.csdn.net/ityard/article/details/102646738'</span>
<span class="token comment"># url = 'http://www.goudan11111.com'</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
        <span class="token comment"># 'Accept':</span>
'text<span class="token operator">/</span>html<span class="token punctuation">,</span>application<span class="token operator">/</span>xhtml<span class="token operator">+</span>xml<span class="token punctuation">,</span>application<span class="token operator">/</span>xml<span class="token punctuation">;</span>q<span class="token operator">=</span><span class="token number">0.9</span><span class="token punctuation">,</span>image<span class="token operator">/</span>webp<span class="token punctuation">,</span>image<span class="token operator">/</span>apng<span class="token punctuation">,</span><span class="token operator">*</span><span class="token operator">/</span><span class="token operator">*</span><span class="token punctuation">;</span>q<span class="token operator">=</span><span class="token number">0.8</span><span class="token punctuation">,</span>applicati
on<span class="token operator">/</span>signed‐exchange<span class="token punctuation">;</span>v<span class="token operator">=</span>b3'<span class="token punctuation">,</span>
        <span class="token comment"># 'Accept‐Encoding': 'gzip, deflate, br',</span>
        <span class="token comment"># 'Accept‐Language': 'zh‐CN,zh;q=0.9',</span>
        <span class="token comment"># 'Cache‐Control': 'max‐age=0',</span>
        <span class="token comment"># 'Connection': 'keep‐alive',</span>
        <span class="token string">'Cookie'</span><span class="token punctuation">:</span> 'uuid_tt_dd<span class="token operator">=</span>10_19284691370‐<span class="token number">1530006813444</span>‐<span class="token number">566189</span><span class="token punctuation">;</span>
smidV2<span class="token operator">=</span><span class="token number">2018091619443662be2b30145de89bbb07f3f93a3167b80002b53e7acc61420</span><span class="token punctuation">;</span>
_ga<span class="token operator">=</span>GA1<span class="token punctuation">.</span><span class="token number">2.1823123463</span><span class="token number">.1543288103</span><span class="token punctuation">;</span> dc_session_id<span class="token operator">=</span>10_1550457613466<span class="token punctuation">.</span><span class="token number">265727</span><span class="token punctuation">;</span>
acw_tc<span class="token operator">=</span><span class="token number">2760821d15710446036596250e10a1a7c89c3593e79928b22b3e3e2bc98b89</span><span class="token punctuation">;</span>
Hm_lvt_e5ef47b9f471504959267fd614d579cd<span class="token operator">=</span><span class="token number">1571329184</span><span class="token punctuation">;</span>
Hm_ct_e5ef47b9f471504959267fd614d579cd<span class="token operator">=</span><span class="token number">6525</span><span class="token operator">*</span><span class="token number">1</span><span class="token operator">*</span>10_19284691370‐<span class="token number">1530006813444</span>‐<span class="token number">566189</span><span class="token punctuation">;</span>__yadk_uid<span class="token operator">=</span>r0LSXrcNYgymXooFiLaCGt1ahSCSxMCb<span class="token punctuation">;</span>
Hm_lvt_6bcd52f51e9b3dce32bec4a3997715ac<span class="token operator">=</span><span class="token number">1571329199</span><span class="token punctuation">,</span><span class="token number">1571329223</span><span class="token punctuation">,</span><span class="token number">1571713144</span><span class="token punctuation">,</span><span class="token number">1571799968</span><span class="token punctuation">;</span>acw_sc__v2<span class="token operator">=</span><span class="token number">5dafc3b3bc5fad549cbdea513e330fbbbee00e25</span><span class="token punctuation">;</span> firstDie<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">;</span> SESSION<span class="token operator">=</span><span class="token number">396bc85c</span>‐<span class="token number">556b</span>‐<span class="token number">42bd</span>‐<span class="token number">890c</span>‐c20adaaa1e47<span class="token punctuation">;</span> UserName<span class="token operator">=</span>weixin_42565646<span class="token punctuation">;</span> UserInfo<span class="token operator">=</span>d34ab5352bfa4f21b1eb68cdacd74768<span class="token punctuation">;</span>UserToken<span class="token operator">=</span>d34ab5352bfa4f21b1eb68cdacd74768<span class="token punctuation">;</span>UserNick<span class="token operator">=</span>weixin_42565646<span class="token punctuation">;</span>AU<span class="token operator">=</span><span class="token number">7A5</span><span class="token punctuation">;</span>UN<span class="token operator">=</span>weixin_42565646<span class="token punctuation">;</span>BT<span class="token operator">=</span><span class="token number">1571800370777</span><span class="token punctuation">;</span>p_uid<span class="token operator">=</span>U000000<span class="token punctuation">;</span>dc_tos<span class="token operator">=</span>pzt4xf<span class="token punctuation">;</span>Hm_lpvt_6bcd52f51e9b3dce32bec4a3997715ac<span class="token operator">=</span><span class="token number">1571800372</span><span class="token punctuation">;</span>Hm_ct_6bcd52f51e9b3dce32bec4a3997715ac<span class="token operator">=</span><span class="token number">1788</span><span class="token operator">*</span><span class="token number">1</span><span class="token operator">*</span>PC_VC!<span class="token number">6525</span><span class="token operator">*</span><span class="token number">1</span><span class="token operator">*</span>10_192846913701530006813444566189!<span class="token number">5744</span><span class="token operator">*</span><span class="token number">1</span><span class="token operator">*</span>weixin_42565646<span class="token punctuation">;</span>announcement<span class="token operator">=</span><span class="token operator">%</span><span class="token number">257B</span><span class="token operator">%</span>2522isLogin<span class="token operator">%</span><span class="token number">2522</span><span class="token operator">%</span>253Atrue<span class="token operator">%</span><span class="token number">252C</span><span class="token operator">%</span>2522announcementUrl<span class="token operator">%</span><span class="token number">2522</span><span class="token operator">%</span><span class="token number">253A</span><span class="token operator">%</span>2522https<span class="token operator">%</span><span class="token number">253A</span><span class="token operator">%</span><span class="token number">252F</span><span class="token operator">%</span>252Fblogdev<span class="token punctuation">.</span>blog<span class="token punctuation">.</span>csdn<span class="token punctuation">.</span>net<span class="token operator">%</span>252Farticle<span class="token operator">%</span>252Fdetails<span class="token operator">%</span><span class="token number">252F102605809</span><span class="token operator">%</span><span class="token number">2522</span><span class="token operator">%</span><span class="token number">252C</span><span class="token operator">%</span>2522announcementCount<span class="token operator">%</span><span class="token number">2522</span><span class="token operator">%</span><span class="token number">253A0</span><span class="token operator">%</span><span class="token number">252C</span><span class="token operator">%</span>2522announcementExpire<span class="token operator">%</span><span class="token number">2522</span><span class="token operator">%</span><span class="token number">253A3600000</span><span class="token operator">%</span><span class="token number">257D</span>'<span class="token punctuation">,</span>
        <span class="token comment"># 'Host': 'blog.csdn.net',</span>
        <span class="token comment"># 'Referer': 'https://passport.csdn.net/login?code=public',</span>
        <span class="token comment"># 'Sec‐Fetch‐Mode': 'navigate',</span>
        <span class="token comment"># 'Sec‐Fetch‐Site': 'same‐site',</span>
        <span class="token comment"># 'Sec‐Fetch‐User': '?1',</span>
        <span class="token comment"># 'Upgrade‐Insecure‐Requests': '1',</span>
        <span class="token string">'User‐Agent'</span><span class="token punctuation">:</span> 'Mozilla<span class="token operator">/</span><span class="token number">5.0</span> <span class="token punctuation">(</span>Windows NT <span class="token number">10.0</span><span class="token punctuation">;</span> Win64<span class="token punctuation">;</span> x64<span class="token punctuation">)</span> AppleWebKit<span class="token operator">/</span><span class="token number">537.36</span> <span class="token punctuation">(</span>KHTML<span class="token punctuation">,</span> like
Gecko<span class="token punctuation">)</span> Chrome<span class="token operator">/</span><span class="token number">77.0</span><span class="token number">.3865</span><span class="token number">.120</span> Safari<span class="token operator">/</span><span class="token number">537.36</span>'<span class="token punctuation">,</span>
    <span class="token punctuation">}</span>

<span class="token keyword">try</span><span class="token punctuation">:</span>
    request <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>
response <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>urlopen<span class="token punctuation">(</span>request<span class="token punctuation">)</span>
    content <span class="token operator">=</span> response<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf‐8'</span><span class="token punctuation">)</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span>content<span class="token punctuation">)</span>
<span class="token keyword">except</span> urllib<span class="token punctuation">.</span>error<span class="token punctuation">.</span>HTTPError<span class="token punctuation">:</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token number">1111</span><span class="token punctuation">)</span>
<span class="token keyword">except</span> urllib<span class="token punctuation">.</span>error<span class="token punctuation">.</span>URLError<span class="token punctuation">:</span>
    <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token number">2222</span><span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li><li style="color: rgb(153, 153, 153);">18</li><li style="color: rgb(153, 153, 153);">19</li><li style="color: rgb(153, 153, 153);">20</li><li style="color: rgb(153, 153, 153);">21</li><li style="color: rgb(153, 153, 153);">22</li><li style="color: rgb(153, 153, 153);">23</li><li style="color: rgb(153, 153, 153);">24</li><li style="color: rgb(153, 153, 153);">25</li><li style="color: rgb(153, 153, 153);">26</li><li style="color: rgb(153, 153, 153);">27</li><li style="color: rgb(153, 153, 153);">28</li><li style="color: rgb(153, 153, 153);">29</li><li style="color: rgb(153, 153, 153);">30</li><li style="color: rgb(153, 153, 153);">31</li><li style="color: rgb(153, 153, 153);">32</li><li style="color: rgb(153, 153, 153);">33</li><li style="color: rgb(153, 153, 153);">34</li><li style="color: rgb(153, 153, 153);">35</li><li style="color: rgb(153, 153, 153);">36</li><li style="color: rgb(153, 153, 153);">37</li><li style="color: rgb(153, 153, 153);">38</li><li style="color: rgb(153, 153, 153);">39</li></ul></pre>
<hr>
<h3><a name="t11"></a><a id="Handler_299"></a>八、Handler处理器</h3>
<p><img src="https://i-blog.csdnimg.cn/blog_migrate/aa656e9135584250a332df7137e62d50.png" alt="在这里插入图片描述"></p>
<ul><li> <p><strong>基本使用</strong></p> <pre data-index="9" class="set-code-show prettyprint"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request

url <span class="token operator">=</span> <span class="token string">'http://www.baidu.com'</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span> <span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>

reqObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>
handler <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>HTTPHandler<span class="token punctuation">(</span><span class="token punctuation">)</span>
opener <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>build_opener<span class="token punctuation">(</span>handler<span class="token punctuation">)</span>
resp <span class="token operator">=</span> opener<span class="token punctuation">.</span><span class="token builtin">open</span><span class="token punctuation">(</span>reqObj<span class="token punctuation">)</span>
context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li></ul></pre> </li><li> <p><strong>使用代理</strong><br> <img src="https://i-blog.csdnimg.cn/blog_migrate/487e1cc7f5f4240ec917964df0f37f37.png" alt="在这里插入图片描述"></p> </li></ul>
<pre data-index="10" class="prettyprint set-code-show"><code class="prism language-python has-numbering" onclick="mdcp.copyCode(event)" style="position: unset;"><span class="token keyword">import</span> random
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>parse
<span class="token keyword">import</span> urllib<span class="token punctuation">.</span>request


<span class="token comment">#随机代理池</span>
proxies_pool <span class="token operator">=</span> <span class="token punctuation">[</span>
    <span class="token punctuation">{<!-- --></span><span class="token string">"http"</span><span class="token punctuation">:</span><span class="token string">'61.163.32.88:312811'</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
    <span class="token punctuation">{<!-- --></span><span class="token string">"http"</span><span class="token punctuation">:</span><span class="token string">'61.163.32.88:3128222'</span><span class="token punctuation">}</span>
 <span class="token punctuation">]</span>

proxies <span class="token operator">=</span> random<span class="token punctuation">.</span>choice<span class="token punctuation">(</span>proxies_pool<span class="token punctuation">)</span>
<span class="token keyword">print</span><span class="token punctuation">(</span>proxies<span class="token punctuation">)</span>


url <span class="token operator">=</span> <span class="token string">'http://www.baidu.com/s?wd=ip'</span>
headers <span class="token operator">=</span> <span class="token punctuation">{<!-- --></span>
    <span class="token string">"User-Agent"</span><span class="token punctuation">:</span> <span class="token string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'</span><span class="token punctuation">,</span>
<span class="token punctuation">}</span>

reqObj <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>Request<span class="token punctuation">(</span>url<span class="token operator">=</span>url<span class="token punctuation">,</span>headers<span class="token operator">=</span>headers<span class="token punctuation">)</span>
handler <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>ProxyHandler<span class="token punctuation">(</span>proxies<span class="token operator">=</span>proxies<span class="token punctuation">)</span>
opener <span class="token operator">=</span> urllib<span class="token punctuation">.</span>request<span class="token punctuation">.</span>build_opener<span class="token punctuation">(</span>handler<span class="token punctuation">)</span>
resp <span class="token operator">=</span> opener<span class="token punctuation">.</span><span class="token builtin">open</span><span class="token punctuation">(</span>reqObj<span class="token punctuation">)</span>
context <span class="token operator">=</span> resp<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">"utf-8"</span><span class="token punctuation">)</span>

<span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">"代理.html"</span><span class="token punctuation">,</span><span class="token string">'w'</span><span class="token punctuation">,</span>encoding<span class="token operator">=</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fp<span class="token punctuation">:</span>
    fp<span class="token punctuation">.</span>write<span class="token punctuation">(</span>context<span class="token punctuation">)</span>
<div class="hljs-button {2}" data-title="复制"></div></code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li><li style="color: rgb(153, 153, 153);">18</li><li style="color: rgb(153, 153, 153);">19</li><li style="color: rgb(153, 153, 153);">20</li><li style="color: rgb(153, 153, 153);">21</li><li style="color: rgb(153, 153, 153);">22</li><li style="color: rgb(153, 153, 153);">23</li><li style="color: rgb(153, 153, 153);">24</li><li style="color: rgb(153, 153, 153);">25</li><li style="color: rgb(153, 153, 153);">26</li><li style="color: rgb(153, 153, 153);">27</li><li style="color: rgb(153, 153, 153);">28</li></ul></pre>
                </div>